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ABSTRACT 

The  Susceptibility  Model  Assessment  and  Range  Test  (SMART)  Project  was  commissioned  in 
FY92  to  develop,  test  and  transition  to  DoD  a  proven  and  efficient  credibility  assessment  process 
for  joint-use  aircraft  survivability  models  and  simulations  (M&S)  currently  supporting  major 
weapons  system  acquisition  and  testing  decisions.  SMART  is  sponsored  by  the  Joint  Technical 
Coordinating  Group  for  Aircraft  Survivability  (JTCG/AS),  is  funded  by  OUSD(A&T)/DT&E, 
and  enjoys  tri-service  participation  and  support.  It  is  scheduled  for  completion  in  FY95. 

SMART  integrates  the  key  elements  of  M&S  credibility  (verification  and  validation  (V&V),  and 
configuration  management  (C/M))  into  a  process  that  provides  essential  information  to  decision¬ 
makers  and  analysts  to  support  accreditation  decisions  for  survivability  M&S. 

This  paper  describes  the  development  history  of  the  SMART  V&V  and  C/M  processes  for  these 
M&S,  and  the  integration  of  this  process  into  an  attack  on  the  accreditation  problem.  Although 
focused  on  V&V  and  C/M  process  development  for  mature  aircraft  survivability  M&S,  the 
approach  taken  and  the  lessons  learned  should  be  of  broad  interest  to  all  who  struggle  with  the 
M&S  credibility  problem. 


INTRODUCTION 

A  meteoric  rise  in  the  capabilities  of  computer  M&S  to  replicate  complex  phenomena,  coupled 
with  a  precipitous  drop  in  defense  outlays,  has  made  M&S  an  extremely  attractive  alternative  to 
costly  testing  in  the  weapons  system  acquisition  process.  M&S  are  used  in  nearly  all  phases  of 
the  acquisition  cycle  (figure  1),  from  concept  development  to  operation  and  maintenance.  The 
high  priority  placed  on  the  war  fighting  capabilities  of  the  few  systems  affordable  in  an  austere 
defense  environment  has  resulted  in  an  ever  more  urgent  requirement  to  ascertain  the  credibility 
of  those  survivability  M&S  that  support  their  purchase. 


Report  Documentation  Page 


Report  Date 

Report  Type 

Dates  Covered  (from...  to) 

00Aug2001 

N/A 

- 

Title  and  Subtitle 

A  "Smart"  Approach  To  VV&A 


Author(s) 

Paul  Muessig 


Performing  Organization  Name(s)  and  Address(es) 

Naval  Air  Warfare  Center,  Weapons  Division  (Code 
418100D)  China  Lake,  California  93555-6001 

Sponsoring/Monitoring  Agency  Name(s)  and 
Address(es) 

Distribution/ Availability  Statement 

Approved  for  public  release,  distribution  unlimited 

Supplementary  Notes 

Abstract 

see  report 


Subject  Terms 


Report  Classification 

unclassified 

Classification  of  this  page 

unclassified 

Classification  of  Abstract 

unclassified 

Limitation  of  Abstract 

SAR 

Contract  Number 
Grant  Number 
Program  Element  Number 
Project  Number 
Task  Number 
Work  Unit  Number 

Performing  Organization  Report  Number 

Sponsor/Monitor’s  Acronym(s) 
Sponsor/Monitor’s  Report  Number(s) 


Number  of  Pages 

16 


Figure  1:  M&S  Use  in  the  Acquisition  Cycle 


It  was  into  this  environment,  and  to  satisfy  this  need,  that  the  SMART  Project  was  commissioned 
in  FY92.  It's  goal  has  been  to  develop,  test  and  transition  to  DoD  an  efficient  credibility 
assessment  process  for  joint-use  aircraft  survivability  M&S.  The  models  selected  to  develop  and 
test  the  credibility  assessment  process  are  well  known  survivability  analysis  models:  the 
Enhanced  Surface  to  Air  Missile  Simulation  (ESAMS);  the  Advanced  Low  Altitude  Radar 
Model  (ALARM);  the  Radar  Directed  Gun  System  Simulation  (RADGUNS);  the  Trajectory 
Analysis  Program  (TRAP);  and  the  Air  to  Air  System  Performance  Evaluation  Model 
(AASPEM).*  These  models  were  chosen  by  a  tri-service  executive  level  "Senior  Steering 
Group"  because  each  of  them  was  (and  is)  currently  used  to  support  major  air  weapons  system 
acquisition  and  testing  decisions  across  the  services. 


ESSENTIALS  OF  M&S  CREDIBILITY 

Over  the  last  decade  or  so,  it  has  become  conventional  wisdom  that  the  key  elements  of  M&S 
credibility  are  verification  and  validation  (V&V),  configuration  management  (C/M),  and 
accreditation.  A  few  years  ago,  the  arcane  workings  of  "VV&A,"  and  the  supporting  functions  of 
C/M,  were  understood  only  by  software  engineers.  Today,  the  term  is  on  the  lips  of  senior  policy 
makers  and  program  managers  across  the  services  and  DoD.  Much  of  the  credit  for  this  is  due  to 
the  Military  Operations  Research  Society  (MORS).  In  it's  laudable  Simulation  Validation 
(SIMVAL)  program,  the  following  definitions  were  proposed  for  these  terms: 

Verification:  The  process  of  determining  that  a  model  implementation  accurately  represents 
the  developer's  conceptual  description  and  specifications. 

Validation:  The  process  of  determining  the  degree  to  which  a  model  is  an  accurate 
representation  of  the  real  world  from  the  perspective  of  the  intended  uses  of  the  model. 

Configuration  Management:  A  discipline  applying  technical  and  administrative  oversight 
and  control  to  identify  and  document  the  functional  requirements  and  capabilities  of  a  model, 
control  changes  to  those  capabilities,  and  document  and  report  the  changes. 


Accreditation:  An  official  determination  that  a  model  is  acceptable  for  a  specific  purpose. 

Although  there  is  still  disagreement  as  to  the  practical  meaning  of  these  terms,  the  MORS 
definitions  have  provided  a  convenient  focal  point  for  V&V  and  C/M  process  development  in  the 
SMART  Project,  which  is  really  the  first  attempt  to  put  them  to  practical  use  on  a  large  scale. 

SMART  integrates  the  "arts"  of  verification,  validation  and  configuration  management  (figure  2) 
into  a  "science"  that  provides  essential  information  to  decision-makers  and  analysts  who  must 
accredit  survivability  M&S  for  use  in  weapons  system  acquisition  and  testing.  The  development 
history  of  the  individual  pieces  of  this  process  is  instructive  for  it's  insights  into  what  constitutes 
M&S  "credibility"  on  the  practical  level,  and  how  credibility  relates  to  "accreditation." 

y&y  process  development 

It  is  impossible  to  appreciate  the  contributions  that  SMART  has  made  to  the  development  of 
V&V  and  C/M  processes  for  survivability  M&S  without  understanding  the  M&S  environment  in 
which  SMART  found  itself  when  it  began  it's  Herculean  (and  occasionally  Sisyphean)  task.  The 
following  sections,  therefore,  define  the  purpose  of  each  piece  of  the  credibility  assessment 
process,  describe  the  unique  challenges  SMART  faced  in  designing  and  applying  a  V&V  process 
to  mature  M&S,  and  conclude  with  a  summary  of  progress  in  each  area.  We  will  then  turn  our 
attention  to  applying  V&V  and  C/M  to  the  pursuit  of  the  Holy  Grail:  accreditation. 


ACCREDITATION 


Figure  2:  Accreditation  and  M&S  Credibility 


Verification 

Why  Verify?  Verification  establishes  that  software  design  specifications  and  requirements  have 
been  identified,  documented,  and  compared  with  their  implementation  in  the  code.  This  process 
flushes  out  design  and  coding  errors  early  in  the  M&S  development  cycle,  making  them  more 
cost  effective  to  repair.  Verification  provides  documented  confidence  in  code  accuracy,  and  in 
M&S  design,  assumptions,  limitations  and  constraints. 


Verification  Challenges.  Although  reasonably  straightforward  for  M&S  in  development,  the 
attempt  to  develop  a  verification  process  for  mature  M&S  encountered  four  distinct  challenges: 

1.  The  codes  for  mature  M&S  were  developed  before  detailed  software  design,  coding  and 
verification  standards  were  in  place.  The  issue  of  which  of  the  myriad  current  standards 
should  apply,  and  how  they  should  be  applied,  to  mature  M&S  therefore  became 
paramount. 

2.  Among  M&S  users,  there  was  a  poor  understanding  of  the  verification  process  in  general, 
and  no  understanding  of  its  technical  elements.  Developing  consensus  among  users  as  to 
the  technical  constituents  of  verification,  therefore,  became  difficult. 

3.  There  was  poor  documentation  of  prior  verification  efforts.  This  made  it  difficult  to  avoid 
reinventing  the  wheel  in  some  cases.  Coupled  with  an  unclear  definition  of  verification  as 
above,  it  became  clear  that  the  documentation  of  technical  results  and  the  development  of 
a  corporate  memory  of  M&S  credibility  assessments  should  be  one  of  the  central  tenets 
of  SMART  Project  philosophy. 

4.  There  was  (and  is)  a  profound  fear  of  cost.  Many  in  the  M&S  community  had  heard  the 
beat  of  distant  drums  from  the  software  development  community  about  verification,  and 
the  drums  tapped  out  "bring  da  money"  to  a  lively  rumba  beat.  This  led  to  a  tendency  to 
avoid  the  verification  problem  altogether  by  redefining  verification  in  such  a  way  as  to 
reduce  its  relevance  to  the  problem  of  M&S  credibility,  thereby  exacerbating  challenge 
number  2. 

The  SMART  Response.  Undaunted  at  (or  ignorant  of)  the  prospect  of  reprising  the  Twelve 
Labors  of  Hercules,  the  SMART  Project  undertook  to  develop  tailored  verification  plans  for 
three  of  its  five  models:  ESAMS,  ALARM  and  RADGUNS  (figure  3).  In  doing  so,  SMART 
pioneered  the  process  of  "verification  in  reverse,"  and  tailored  this  process  to  mature, 
survivability  M&S.  We  did  this  by  identifying  reasonable  verification  standards  in  light  of  DoD, 
MIL  and  service  standards  for  software  verification,  and  guidance  from  each  model's 
developmental  and  VV&A  history.  Automated  technologies  to  support  the  process,  such  as 
Computer  Aided  Software  Engineering  (CASE)  tools,  were  also  investigated  and  applied  where 
appropriate. 
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Figure  3:  Verification  Process  Development 


The  goal  of  reviewing  existing  service,  DoD  and  MIL  standards  (e.g.,  2167A,  2168,  1521B, 
488A)  was  to  seek  guidance  on  the  software  verification  process,  to  specify  which  verification 
steps  were  essential  to  the  credibility  of  mature  M&S,  and  to  identify  a  minimum  set  of 
documentation  that  would  establish  the  credibility  of  a  mature  M&S  from  the  verification  stand¬ 
point.  Five  pieces  of  documentation  were  identified  as  essential  to  M&S  usage  and  verification:  a 
Software  User's  Manual  (SUM),  Programmer's  Manual  (SPM),  Analyst's  Manual  (SAM),  Design 
Document  (SDD)  and  Verification  Report.  Of  these,  the  three  models  in  question  had  the  first 
three  documents  in  some  form  or  another.  The  SDD,  however,  without  which  detailed 
verification  would  be  an  ignis  fatuus,  did  not  exist  for  any  model;  likewise  the  Verification 
Report.  SMART,  therefore,  pioneered  a  post-development  substitute  for  the  SDD  (required  by 
2167A),  appropriately  named  the  Post-Development  Design  Document  (PDDD).  The  PDDD  is 
the  basis  for  all  detailed  verification  of  code. 

To  capitalize  on  prior  work  and  experience  in  verification,  we  surveyed  ESAMS,  ALARM  and 
RADGUNS  users  and  developers  to  establish  development,  V&V  and  C/M  statuses  and 
histories.  Experience  confirmed  what  reason  suggested:  a  dismal  (but  not  wholly  unanticipated) 
state  of  basic  information  relating  to  the  history  of  model  development  and  credibility 
assessment.  This  substantiated  the  need  for  SMART  to  do  something  about  the  lack  of 
documentation  supporting  M&S  credibility. 

Having  identified  what  should  be  done  and  what  had  been  done,  we  proceeded  to  assess  software 
quality  using  CASE  tools.  Factors  such  as  use  of  standards,  adherence  to  programming 
conventions,  computational  efficiency  and  memory  utilization  were  assessed  in  order  to  quantify 
the  relationship  between  code  quality  and  verification  efficiency.  With  this  last  piece  in  place,  we 
developed  tailored  verification  plans  for  the  subject  models. 

Finally,  to  address  fear  of  cost,  SMART  conducted  a  Pilot  Verification  Study  on  the  three  subject 
models.  Using  the  tailored  plans  as  a  guide,  the  four  highest  priority  common  model  functions 
(based  on  sensitivity  analysis;  see  below)  were  verified.  The  goal  was  to  identify  the  cost 
implications  of  performing  "verification  in  reverse,"  and  to  investigate  alternative  methods  of 
verification.  Results  of  this  study  are  scheduled  for  publication  at  the  end  of  FY94. 


Progress  to  Date.  In  addiition  to  documentation  of  the  studies  described  above,  PDDDs  have 
been  developed  for  ESAMS,  ALARM  and  RADGUNS  which  provide  detailed  software  design 
specifications  for  these  models.  Verification  Reports  are  also  available,  which  provide  the  results 
of  detailed  software  testing  and  desk  checking  for  each  model  at  the  function  level.  A 
verification  process  paper  providing  details  of  process  elements  and  recommendations  for 
application  to  other  M&S  is  scheduled  for  publication  at  the  end  of  FY94.  Verification  efforts  for 
TRAP  and  AASPEM  are  scheduled  to  begin  in  FY95,  if  resources  are  available.* 

Validation  Why  Validate?  The  fundamental  impetus  for  validation  is  a  fiscal  one;  it  is  cheaper 
to  compute  than  to  test.  Validation  of  M&S  permits  the  replacement  of  a  measurement  (test)  with 
a  prediction  about  a  measurement  (computation).  Validation  provides  confidence  that  a  model 
behaves  like  the  "real  world"  (assumed  to  be  well  defined),  at  least  within  certain  specified 
boundaries  (also  assumed  to  be  well  defined).  If  we  know  the  domain  of  conditions  over  which 
validation  was  performed,  and  the  correlation  between  prediction  and  observation  within  that 
domain,  we  can  determine  whether  the  model  behaves  enough  like  the  "real  world"  for  a  given 
application  by  defining  criteria  for  acceptable  correlation.  But  we  must  have  all  three  pieces  of 
information:  validation  boundaries,  correlation  results  and  acceptance  criteria. 

Validation  Challenges.  SMART  faced  three  distinct  challenges  in  developing  a  validation 
process  for  mature  M&S: 

1.  Functional  redundancies  across  M&S  were  neither  identified  nor  exploited.  The  resultant 
"model  by  model"  validation  paradigm  meant  that  the  same  data  were  being  collected 
multiple  times  for  validating  the  same  functions  of  different  models.  If  functional 
redundancies  across  M&S  could  be  identified  and  exploited,  a  few  well  focused  data 
collection  efforts  could,  in  theory,  collect  enough  data  to  validate  large  parts  of  multiple 
models  over  a  reasonably  broad  domain. 

2.  Dedicated,  model  by  model  validation  testing  was  (and  is)  simply  too  costly.  (Fear  of  cost 
again).  For  SMART  to  fulfill  its  mission  to  develop  a  cost-efficient  approach  to  M&S 
credibility  across  service  boundaries,  it  could  not  squander  limited  resources  on  massive, 
redundant  data  collection  efforts  and  dedicated  testing.  The  return  on  investment  would 
be  too  low.  An  alternative,  high  volume  source  of  test  data  had  to  be  found. 

3.  Documentation  of  prior  validation  efforts  was  sparse  or  nonexistent.  Although  many  in 
the  M&S  community  claimed  they  had  been  doing  validation  "for  years,"  when  pressed 
for  documented  results,  from  which  SMART  could  develop  an  integrated  approach,  it 
became  apparent  that  years  of  work  would  be  irretrievably  lost  with  the  retirement  of  the 
current  generation  of  analysts.  Again,  it  was  clear  that  the  issue  of  corporate  memory  was 
of  paramount  importance. 

The  SMART  Response.  Figure  4  encapsulates,  in  highly  stylized  form,  the  essence  of  the 
SMART  validation  process,  developed  in  response  to  these  challenges. 


Figure  4:  Validation  Process 


In  order  to  avoid  costly,  repetitive,  dedicated  data  collection  efforts,  SMART  decomposes  M&S 
into  functional  elements  (FE's),  paying  careful  attention  to  identifying  common  functions  across 
them.  It  then  distills  the  results  of  the  decomposition  effort  into  hierarchical  functional  element 
templates,  showing  both  the  common  and  the  unique  functions  of  the  models.  These  templates 
form  the  basis  from  which  other  models  can  be  decomposed  and  functional  similarities 
identified. 

For  each  FE  identified,  a  two  phase  sensitivity  analysis  is  conducted  for  each  model.  Phase  I 
sensitivity  analysis  identifies  those  model  functions  that  have  the  greatest  impact  on  top  level 
model  outputs  (e.g.,  detection  range  for  AFARM,  probability  of  kill  for  ESAMS,  and  probability 
of  hit  for  RADGUNS).  The  purpose  is  to  prioritize  which  functions  need  validation  first. 
Functions  with  marginal  impact  on  top  level  outputs  become  candidates  for  "face  validation"  by 
a  panel  of  subject  matter  experts  (SME's),  saving  the  trouble  (and  the  cost)  of  explicit  data 
collection  for  validation  of  these  functions.  For  FE's  with  a  large  impact  on  top  level  results, 
Phase  II  sensitivity  analysis  is  conducted  at  the  function  level,  the  purpose  of  which  is  to  identify 
highly  sensitive  data  elements  within  each  function,  and  to  specify  data  collection  requirements 
for  them  (e.g.,  required  accuracies  and  sampling  rates).  The  resulting  list  of  data  requirements  for 
each  function  across  M&S  is  consolidated  into  a  Data  Requirements  Dictionary,  forming  the 
basis  for  the  development  of  a  library  of  notional  test  plans  for  each  FE.  These  test  plans  are  not 
specific  to  any  model,  but  derive  their  particulars  from  analysis  of  all  models  containing  the 
given  function. 

Armed  with  data  collection  requirements  and  plans,  finding  economical  sources  of  data  is 
paramount.  SMART  attacked  this  problem  by  conducting  an  in  depth  review  of  tri-service 
development  test  (DT),  operational  test  (OT),  foreign  materiel  exploitation  (FME),  laboratory 
and  bench  test  programs,  seeking  those  with  promise  of  providing  the  data  specified  in  the  DRD 
in  accordance  with  the  specifications  outlined  in  the  notional  test  plans.  Where  possible, 
Memoranda  of  Understanding  (MOUs)  between  SMART  and  relevant  test  agencies  and 
programs  were  negotiated.  These  MOU's  provide  access  not  only  to  test  data  after  the  test,  but 
access  to  the  test  planning  process  itself.  This  allows  SMART'S  data  collection  requirements  to 
be  inserted  early  enough  in  the  test  cycle  to  impact  data  collection  objectives.  Where  additional 


data  can  be  collected  to  support  model  validation  objectives,  SMART  pays  for  the  incremental 
cost  under  the  auspices  of  the  larger  test  program.  In  this  way,  the  high  costs  of  range  time,  test 
platform  support,  system  calibration,  etc.  are  avoided,  while  still  collecting  a  high  volume  of 
data. 

With  data  collection  objectives  specified,  and  access  to  test  data  guaranteed,  SMART  developed 
an  incremental  approach  to  comparing  test  data  with  model  predictions.  Before  detailed 
comparisons  with  test  data  are  attempted,  system  characterization  and  calibration  data  from  a  test 
article  (e.g.,  a  radar  or  a  missile)  and  system  performance  as  modeled  in  the  code  are  compared. 
For  example,  a  missile  system's  tracking  radar  may  have  technical  characteristics  not  modeled  in 
ESAMS  (say)  because  data  to  support  such  modeling  were  unavailable  at  the  time  ESAMS  was 
developed.  In  this  case,  the  test  article  and  the  model's  simulation  of  the  test  article  may  be 
profoundly  different.  Some  attempt  must  be  made  to  reconcile  difference  between  the  two,  lest 
comparisons  with  field  test  data  be  meaningless.  In  some  cases,  the  model  is  modified  to 
incorporate  system  features  or  capabilities.  In  other  cases  it  is  corrected  on  the  basis  of  system 
data.  In  all  cases,  identification  of  discrepancies  between  hardware  and  model  are  identified,  as 
well  as  their  possible  impact  on  the  conclusions  of  the  validation  effort.  Comparison  of  function 
level  test  results  with  model  predictions  lead  to  Functional  Element  Assessment  Reports  (or, 
rather  prophetically,  FEARs). 

Comparisons  between  field  test  data  and  model  predictions  are  done  by  an  independent  agent.  In 
the  case  of  ESAMS,  the  independent  validator  is  the  RAND  Corporation,  one  of  the  Air  Force's 
primary  ESAMS  users,  and  affiliated  with  ESAMS  sponsor  organization,  the  Air  Force  Studies 
and  Analysis  Agency  (AFSAA).  The  comparison  between  model  predictions  and  field  test 
observations  flushes  out  modeling  and  coding  errors  that  result  in  Model  Deficiency  Reports 
(MDR's).  These  are  transmitted  to  the  model  developer  for  consideration  under  the  configuration 
management  cycle. 

Comprehensive  documentation  of  the  technical  details  of  validation  results  at  both  the  function 
level  and  the  model  level  is  contained  in  the  Validation  Report,  which  becomes  part  of  an 
expanding  archive  of  model  assessment  information.  Test  data  and  their  supporting 
documentation  are  also  archived,  providing  a  body  of  well  documented  validation  data  that  may 
be  of  use  to  future  users.  Both  the  Validation  Report  and  the  Test  Data  Archive  should  be 
available  at  the  end  of  FY94. 

Progress  to  Date.  To  date,  Functional  Element  Assessment  Reports  (or,  rather  prophetically, 
FEARs)  for  nearly  all  radar  functional  elements  for  ESAMS,  ALARM  and  RADGUNS  have 
been  produced.  Validation  of  some  missile  functions  in  ESAMS  and  some  AAA  functions  in 
RADGUNS  are  also  included  in  these  reports.  These  reports  include  functional  element 
descriptions,  sensitivity  analysis  results,  data  collection  requirements,  and  validation  results. 
Data  collection  efforts  for  the  last  two  years  of  the  program  will  focus  on  expanding  the 
validation  space  for  each  of  these  models,  and  beginning  validation  work  for  TRAP  and 
AASPEM. 

Configuration  Management 

Why  Manage  a  Model's  Configuration?  Configuration  Management  is  a  user  support  function. 
A  government  agent  (typically  referred  to  as  the  "model  developer")  manages  the  process  of 


model  change  in  an  orderly,  auditable  way.  C/M  functions  support  users  with  orderly 
development,  consistency  of  results  across  model  versions,  version  control,  timely 
documentation,  and  error  identification,  resolution  and  tracking.  The  result  of  all  this  should  be 
the  ability  to  compare  model  results  across  versions,  and  to  prevent  (or  at  least  inhibit)  version 
proliferation  by  keeping  the  model  and  its  documentation  as  up  to  date  and  useful  as  possible. 

Configuration  Management  Challenges.  No  model  user  can  resist  the  temptation  to  tinker  with 
a  model,  however,  even  a  good  one.  (Are  there  any  "good"  models,  seen  through  an  analyst's 
eyes?)  Either  the  model  does  not  perform  all  the  functions  the  user  requires,  or  else  it  does  not 
model  these  functions  in  sufficient  detail  for  a  particular  application,  or  else  something.  There  is 
always  some  reason  why  a  model  absolutely,  positively  has  to  be  changed,  right  now,  from  its 
pristine,  configuration  managed  state.  As  a  result,  there  is  a  natural  tension  between  model  users 
and  model  developers. 

What  keeps  this  tension  in  balance  is  the  "leaven"  in  the  C/M  bread:  money.  Model  developers 
must  prioritize  suggested  changes  (and  occasionally,  even  error  corrections)  within  the  confines 
of  a  limited  budget,  although  users  occasionally  bring  money  to  the  C/M  table  to  influence 
change  priorities.  If  the  change  priorities  that  emerge  from  this  resource  constrained  environment 
do  not  meet  market  (user)  needs,  the  impetus  for  model  changes  outside  the  C/M  process 
increases.  The  result  is  fragmented  development,  inconsistent  results  across  model  versions, 
version  proliferation,  poor  documentation,  and  unreliable  error  tracking:  in  short,  the  exact 
opposite  of  C/M  goals. 

The  SMART  Response.  Into  this  user-developer  dynamic,  SMART  has  inserted  itself  with  a 
singular  vision:  the  remediation  of  C/M  woes  via  the  development  of  consistent  C/M 
requirements  and  process  guidelines  across  M&S.  We  believe  that  consistent  standards  for  C/M 
are  the  first  step  toward  consistent  implementation  and  results;  conversely,  we  believe  that  the 
lack  of  standards  by  which  to  judge  a  C/M  process  or  its  results  leads  to  inconsistent  application 
of  other-wise  laudable  C/M  principles,  and  enormously  complicates  an  already  complicated  task. 
Figure  5  shows  the  process  by  which  SMART  has  approached  this  task. 

We  began  by  analyzing  current  C/M  practices  across  ESAMS,  ALARM  and  RADGUNS, 
looking  for  similarities  and  differences,  strengths  and  weaknesses.  So  as  to  heed  the  proverbial 
admonition  not  to  rely  upon  our  own  understanding,  we  also  conducted  user  and  model 
developer  surveys  to  identify  what  these  groups  thought  was  good  and  bad  about  current  C/M 
practices  for  their  models,  and  what  they  wanted  to  see  changed.  We  analyzed  and  distilled  this 
information  into  a  set  of  preliminary  C/M  process  requirements  and  recommendations,  and 
briefed  them  to  the  model  developers  for  consensus  and  detailed  definition.  We  have  now  set  up 
a  cycle  of  regular  meetings  with  model  managers  and  users  to  codify  C/M  requirements  and 
definitions  across  ESAMS,  ALARM  and  RADGUNS,  with  the  ultimate  goal  of  incorporating 
them  into  a  set  of  consistent  C/M  process  guidelines  that  can  be  applied  to  other  M&S. 

Progress  to  Date.  Recommendations  for  improvement  based  on  the  work  described  above  have 
been  incorporated  into  the  C/M  process  for  one  model  (ESAMS).  The  recommendations  attack 
chronic  problems  of  version  proliferation,  inconsistent  documentation,  user  communication,  and 
the  time  it  takes  to  update  the  ESAMS  baseline.  The  trial  run  of  these  revised  practices  will  be  in 
place  for  one  year,  and  is  being  monitored  for  lessons  learned  that  will  influence  the 
development  of  final  C/M  requirements  and  process  guidelines,  with  possible  extension  to  model 


management  for  other  models  in  the  SMART  set. 

Work  has  also  begun  on  documenting  the  results  of  our  C/M  Requirements  Study.  This 
document  will  include  a  summary  of  C/M  practices  for  the  three  subject  models,  an  analysis  of 
the  change  flow  process  inherent  in  each,  and  a  description  of  the  advantages  and  disadvantages 
of  each  approach.  The  document  will  also  include  an  analysis  of  user  and  model  manager 
surveys,  a  listing  of  agreed  upon  C/M  requirements  and  principles  across  the  three  M&S,  and  a 
list  of  recommended  C/M  process  enhancements.  The  document  is  intended  to  be  a  springboard 
for  the  development  of  integrated  C/M  policies,  procedures  and  guidelines  for  survivability  M&S 
that  should  have  broad  relevance  to  other  classes  of  M&S.  This  document  is  scheduled  for 
publication  in  late  FY94. 


Figure  5:  C/M  Requirements  Study 


RELEVANCE  TO  ACCREDITATION 

Why  Accredit?  Accreditation  is  the  summum  bonum,  the  "Holy  Grail,"  of  M&S  credibility 
efforts,  focusing  the  complex  technical  tasks  described  above  on  the  very  practical  question  of 
whether  or  not  a  model  is  "good  enough  for  government  work."  Without  a  formal  stamp  of 
approval  from  an  accrediting  authority,  M&S  results  in  support  of  testing  and  analysis  will 
always  be  suspect.  One  would  think  it  axiomatic,  therefore,  given  our  prior  discussion  of  the 
components  of  M&S  credibility,  that  accreditation  demands  satisfaction  of  a  well  defined  set  of 
acceptance  criteria,  criteria  which  would  include  the  requirement  for  V&V  and  C/M.  This  has 
not  typically  been  the  case,  however. 

Accreditation  Challenges.  Until  very  recently,  accreditation  by  acclaim  (or  fiat)  was  more  or 
less  the  rule.  If  a  group  of  experts  or  a  large  body  of  users  had  used  a  model  for  a  long  time  and 
felt  comfortable  with  its  results,  and  if  the  study  agent  had  heard  a  lot  about  the  model  and  didn't 
have  many  (or  any)  alternatives,  accreditation  required  no  more  than  a  signature.  Recognition  of 
a  model's  name  was  tantamount  to  accreditation.  This  was  partly  due  to  the  fact  that 
requirements  for  detailed  accreditation  policies  and  procedures  have  only  recently  been  imposed, 
and  the  fact  that  the  accreditation  process  itself  has  been  poorly  understood. 


With  the  promulgation  of  DoD  5000  series  Directives  and  Instructions  relating  to  M&S  over  the 
last  two  years  (especially  the  most  recent  DoD  Directive  5000.59),  more  emphasis  has  been 
placed  on  accreditation  process  development  and  execution  across  the  services.  The  Army  was 
the  first  to  respond  comprehensively,  with  its  Army  Regulation  5-11  (AR  5-11)  and  its 
amplifying  AR  5-11  Pamphlet,  which  specified  accreditation  policies  and  procedures,  and  which 
provided  V&V  guidelines.  The  Navy  is  also  developing  an  OPNAV  Instruction  relating  to  the 
management  of  Navy  M&S,  and  includes  guidelines  on  accreditation  and  V&V.  The  Air  Force 
(particularly  AF/XOM  and  its  subsidiary  organization,  AFSAA)  is  currently  in  the  throes  of 
M&S  policy  development  that  includes  V&V  and  accreditation  guidelines  much  in  keeping  with 
the  lessons  learned  during  the  Army  and  Navy  efforts. 

Although  similar  strains  of  music  are  heard  in  each  of  these  policy  directives,  the  result  is  hardly 
euphonious.  Two  factors  drive  the  apparent  cacophony: 

1.  There  are  no  consistent  accreditation  requirements  or  guidelines  across  the  services, 
making  it  hard  for  one  service  to  use  another  service's  models  without  repeating  similar 
accreditation  steps. 

2.  The  application  of  stated  accreditation  guidelines,  both  across  and  within  the  services,  is 
equally  inconsistent,  essentially  leaving  the  meaning  of  accreditation  in  the  eye  of  the 
beholder. 

The  SMART  Response.  Figure  6  summarizes  the  results  of  SMART'S  attempt  to  identify  a 
consistent  set  of  accreditation  requirements  across  the  services,  and  to  tailor  its  V&V  and  C/M 
products  to  meet  those  requirements. 

SMART  has  developed  a  set  of  accreditation  information  requirements  based  on  emerging 
policies,  procedures  and  guidelines  from  across  the  services  and  DoD,  and  has  divided  its  V&V 
and  C/M  processes  into  increments  (figures  7A-C)  that  produce  the  essential  information 
elements  that  support  accreditation.  This  incremental  process  reduces  the  cost  required  to 
accredit  M&S  by  focusing  V&V  and  C/M  efforts  on  identifiable  information  requirements,  and 
by  making  the  results  available  to  the  wider  M&S  community  via  an  Accreditation  Support 
Database. 
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Figure  6:  Accreditation  Requirements  Study  Results 


SMART'S  Accreditation  Requirements  Study  led  to  four  conclusions  with  broad  ramifications  for 
accreditation  policy  and  practice: 

1.  Current  and  emerging  accreditation  policies  are  burdened  with  excessive  administrative 
overhead,  place  artificial  barriers  between  the  Accreditation  Agent  and  those  performing 
the  work,  and  tend  to  turn  the  accreditation  process  into  a  bureaucratic,  process-oriented, 
"check  the  box"  operation. 

2.  Current  accreditation  practices  focus  on  the  collection  of  essential  information  elements 
that  relate  to  various  facets  of  M&S  credibility,  and  place  the  burden  of  proof  squarely  on 
the  shoulders  of  those  responsible  for  study  results.  The  emphasis  of  accreditation 
practice  is  on  information,  not  process. 

3.  SMART  products  satisfy  the  vast  majority  of  information  requirements  that  support 
current  accreditation  practices. 

4.  The  SMART  process  can  be  incrementalized  to  produce  these  information  elements  in 
accordance  with  the  definitions  of  "levels  of  accreditation"  currently  being  formulated  in 
policy  directives  across  the  services. 

Incremental  Accreditation 

Figures  7A-C  depict  SMART'S  incremental  approach  to  the  production  of  accreditation 
information.  Level  I  Accreditation  (figure  7A)  provides  model  users  with  a  baseline  M&S 
characterization.  For  example,  what  is  the  C/M  baseline  for  this  model,  and  how  are  changes  to  it 
controlled?  What  is  the  status  of  model  documentation,  and  how  well  does  it  support  model  use? 
What  is  the  model's  V&V  and  C/M  status  and  history?  What  are  the  inherent  assumptions, 
limitations  and  errors  which  must  be  taken  into  account  before  further  consideration  of  the  model 
is  warranted?  Level  I  information  will  likely  be  collated  into  a  "Green  Book  Report"  which  will 
provide  enough  information  to  the  prospective  model  user  or  accreditation  agent  to  decide 
whether  to  continue  serious  consideration  of  a  model  for  use  in  a  particular  application. 

Level  II  Accreditation  tasks  (figure  7B)  provides  more  in-depth  assessment  of  a  model.  Results 
of  detailed  sensitivity  analyses  at  both  the  model  and  function  levels  are  reported,  as  well  as  face 
verification  and  face  validation  of  the  model  by  a  panel  of  independent  SME's.  Coupled  with 
Level  I  information,  the  "Yellow  Book  Report"  provides  the  best  possible  characterization  of  the 
model  without  actually  performing  detailed  V&V  efforts. 

Finally,  Level  III  Accreditation  tasks  (figure  7C)  provide  the  most  detailed  information  relating 
to  model  credibility.  At  this  level,  a  comprehensive  verification  effort  is  conducted  on  model 
design,  logic  and  code,  including  software  testing,  desk  checking,  and  source  reporting.  Minor 
errors  are  identified  and  corrected;  major  errors  are  reported  to  users  and  model  developers  for 
action  via  the  C/M  process.  Verification  culminates  in  a  comprehensive  specification  of  model 
design  and  implementation  at  the  subroutine  level.  In  addition,  test  data  from  a  variety  of  sources 
are  compared  to  model  predictions  at  the  function  level  and  the  overall  level  (similar  to 
sensitivity  analysis).  DT,  OT,  laboratory,  bench,  FME  and  S&TI  data  sources  are  all  applied  to 
the  validation  effort  to  develop  as  comprehensive  a  validation  space  as  possible.  In  the  process, 


reduced  data  packages  add  to  the  library  of  information  that  future  users  can  draw  upon  to 
validate  similar  M&S,  and  test  plans  and  test  reports  for  each  data  set  provide  an  audit  trail  of 
data  use.  The  results  are  summarized  in  a  "Red  Book  Report." 

Benefits  of  Incremental  Accreditation 

There  are  numerous  benefits  to  an  incremental  approach  to  accreditation.  First,  accreditation 
becomes  cost  effective  when  future  users  can  rely  on  information  developed  by  prior  model 
users.  Since  Level  I  and  II  data  elements  are  more  or  less  model  specific  (vice  application 
specific),  they  will  have  broad  applicability  to  other  users  with  different  applications  for  the 
same  model.  Moreover,  the  cost  of  keeping  Level  I  and  II  accreditation  information  current  drops 
significantly  once  the  basic  information  is  gathered  and  documented. 
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Figure  7A:  Level  I  Accreditation  Increments 


Second,  the  various  accreditation  levels  lend  themselves  to  certain  "natural"  sponsor  affinities. 
For  example,  the  JTCG/AS  and  the  Survivability  and  Vulnerability  Information  Analysis  Center 
(SURVIAC)  have  a  vested  interest  in  the  development  of  survivability  methodology  and  the 
maintenance  of  survivability  M&S;  these  organizations  would,  therefore,  be  a  logical  sponsor  of 
Level  I  accreditation  efforts.  Similarly,  model  managers  (like  AF/XOM  for  ESAMS)  are 
responsible  for  V&V  planning  and  execution,  and  for  providing  their  respective  user 
communities  with  credible  models  "off  the  shelf."  Building  on  Level  I  information  elements, 
these  organizations  would  be  a  logical  sponsor  of  Level  II  accreditation  efforts.  Finally,  weapons 
programs  have  the  greatest  need  for  detailed  comparisons  of  model  predictions  with  test  data  to 
support  Cost  and  Operational  Effectiveness  Analyses  (COEAs),  Defense  Acquisition  Board 
(DAB)  reviews,  and  so  on.  These  organizations  would  be  the  most  likely  candidates  to  build  on 
Levels  I  and  II,  and  sponsor  Level  III  accreditation  efforts. 


c-h!" 


hCCFEnThVIDH 

SLPnFTT 

ps®ounr 

PFUCESS 

aJFTOPT 

REDUDI 

■.w-D  hthh 
MET!  QIC  NTT 

ehhfvufihthh 

UhMMBJEMT 

IKPEUHTT 

Ti,-h  Okiii- 

ir  i-t  Hi... 

P-d-imC-i  Hii.. 

|.H,L,I  IH-llJ 

P-  i  [■-  u- 1- 1-  m-  ■  i 
D-  H  ■  C1- '  ■n-i  i 

A,L.  1 

4U 

A,L.  1 

“■a:: 

F 

C~  \ 


T-  in-J 
Ll-hi... 


A  -  m-n 
nil  Ll-i-d... 


Fi-  V-J Hi i - 

-i  - 1-  -i  f  c 


Pi-i-ii- 
4-H>  Ex 
■  Hl-hl 
T-  iPhi 


Pl-l-,l- 
FC  EikJ-  4-1 
A  -  m-« 
FHa 


C  i, HI  h  VW 
Pri-iH-  F... 

U  Hil-lh 


Figure  7B:  Level  II  Accreditation  Increments 
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Figure  7C:  Level  III  Accreditation  Increments 


Third,  no  one  sponsor  is  liable  for  the  cost  of  the  entire  V&V  for  a  given  M&S.  By 
incrementalizing  the  accreditation  process,  and  by  relating  the  outputs  of  each  increment  to  the 
level  of  confidence  one  can  place  in  M&S  results,  cost  effective  accreditation  at  varying  levels  of 
detail,  to  suit  varying  applications,  becomes  a  reality. 

Cost 

But  all  this  is  abstract  theorizing.  Cost  is  a  big  issue  in  V&V,  and  has  many  service  organizations 
worried  about  fulfilling  the  requirements  of  the  accreditation  policies  being  developed.  One  of 
the  most  withering  criticisms  of  V&V  in  general,  and  SMART  in  particular,  is  that  it  costs  "too 
much."  This  mentality  can  lead  to  the  emasculation  of  V&V  requirements,  diluting  them  to  the 
point  of  practical  insignificance  and  returning  us  to  the  days  of  accreditation  by  acclaim. 

To  address  the  cost  issue,  SMART  has  used  the  incremental  approach  to  V&V  described  above 


to  estimate  the  cost  of  each  level  of  accreditation,  based  on  the  cost  of  the  technical  tasks 
associated  with  each  increment  in  the  SMART  process.  Figure  8  shows  the  approximate  cost  of 
each  phase  of  accreditation,  based  on  applying  the  SMART  process  to  a  model  equal  in 
complexity  to  ESAMS. 

The  ostensibly  large  costs  of  Level  III  accreditation  are  mitigated  by  the  fact  that  these  efforts 
would  characterize  the  entire  model  with  at  least  one  data  set  for  each  function,  and  would  not  be 
borne  by  a  single  user.  Priorities  for  detailed  V&V  would  be  driven  by  the  user  "market,"  and  the 
costs  over  time  would  essentially  be  split  between  users  requiring  Level  HI  accreditation. 
Moreover,  future  users  would  benefit  from  prior  Level  III  accreditation  efforts,  and  would  only 
have  to  fund  those  V&V  efforts  relating  directly  to  their  application,  which  would  in  turn  benefit 
future  users. 
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Figure  8:  Approximate  Cost  of  Incremental  Accreditation 


Accreditation  Support  Products 

SMART  Project  deliverables  include  not  only  documented  and  tested  V&V  and  C/M  processes, 
but  also  a  series  of  reports  describing  the  verification,  validation  and  C/M  status  of  each  M&S  in 
the  SMART  set.  These  reports  supply  baseline  credibility  assessments  for  mature  survivability 
M&S  that  can  support  accreditation  decisions  across  the  services,  resulting  in  smaller,  non- 
duplicative  efforts.  As  a  result  of  the  Accreditation  Requirements  Study,  SMART  products  are 
being  reformatted  to  support  the  accreditation  levels  defined  by  the  study.  This  reformulation  of 
SMART  products  should  be  complete  by  the  end  of  LY94. 


SUMMARY 

SMART  has  successfully  addressed  major  technical  challenges  in  all  areas  of  M&S  credibility 
assessment.  In  addressing  these  challenges,  SMART  has  developed  and  tested  a  sound  V&V 
process  that  supports  both  user  requirements  for  credible  M&S,  and  service  requirements  for  key 
information  supporting  accreditation  of  these  M&S.  Lewer  and  fewer  skeptical  voices,  who  early 
on  had  "seen  this  type  of  thing  come  and  go  before,"  are  being  heard,  and  a  ground  swell  of 
support  for  the  sound  technical  products  achieved  by  the  SMART  process  is  growing. 

As  a  result,  SMART  is  fundamentally  altering  the  way  M&S  credibility  is  perceived.  The  key 


elements  of  M&S  credibility,  first  articulated  by  MORS,  are  better  understood,  and  the  technical 
ramifications  of  these  definitions  have  been  explored.  Moreover,  the  relationship  between  these 
elements  and  accreditation  is  becoming  much  clearer. 

The  challenge  that  both  SMART  and  the  M&S  community  face  is  twofold: 

1.  Expanding  the  scope  of  survivability  M&S  that  have  undergone  a  SMARTtype 
credibility  assessment  process.  This  will  guarantee  the  utility  of  the  process  to  the  entire 
class  of  survivability  M&S,  and  provide  an  ever-expanding  baseline  of  accreditation 
support  information  for  them. 

2.  Applying  the  lessons  learned  from  the  development  of  the  SMART  process  to  other  types 
(non  survivability)  M&S.  From  component  level  simulations  of  individual  electronic 
circuits  to  integrated  strike  warfare  training  simulators,  credibility  is  paramount  to  the 
development  of  a  rational  basis  on  which  to  save  defense  dollars  using  M&S. 

The  SMART  motto  is  "Credible  Models  for  Credible  Analysis."  We  have  done  much  to  make 
this  a  statement  of  fact,  instead  of  just  a  laudable  goal.  We  need  to  keep  putting  our  money  where 
our  mouth  is. 
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